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ABSTRACT 

Introduction: Duplex ultrasound facilitates bedside diagnosis and hence timely patient care. Its uptake has 
been hampered by training and accreditation issues. We have developed an assessment tool for Duplex arterial 
stenosis measurement for both simulator and patient based training. 

Methods: A novel assessment tool: duplex ultrasound assessment of technical skills was developed. A modified 
duplex ultrasound assessment of technical skills was used for simulator training. Novice, intermediate experi- 
ence and expert users of duplex ultrasound were invited to participate. Participants viewed an instructional 
video and were allowed ample time to familiarize with the equipment. Participants' attempts were recorded and 
independently assessed by four experts using the modified duplex ultrasound assessment of technical skills. 
'Global' assessment was also done on a four point Likert scale. Content, construct and concurrent validity as 
well as reliability were evaluated. 

Results: Content and construct validity as well reliability were demonstrated. The simulator had good satisfac- 
tion rating from participants: median 4; range 3-5). Receiver operator characteristic analysis has established a 
cut point of 22/34 and 25/40 were most appropriate for simulator and patient based assessment respectively. 
We have validated a novel assessment tool for duplex arterial stenosis detection. Further work is underway to 
establish transference validity of simulator training to improved skill in scanning patients. 
Conclusions: We have developed and validated duplex ultrasound assessment of technical skills for simulator 
training. 
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INTRODUCTION 

Vascular ultrasound training. On both sides 
of the Atlantic, there is accumulating data 
regarding excessive patients mortality at 
weekends where full hospital care provi- 
sion is not offered (1). Duplex scanning for 
vascular patients is currently performed 
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largely by specialists, varyingly known as 
vascular sonographers/ sonologists/ scien- 
tists/ technologists. Again, on both sides of 
the Atlantic, there is a chronic shortage of 
vascular sonographers (2) and indeed in 
the UK is listed as a shortage occupation by 
the UK Borders agency (3). It seems very 
unlikely that a 24 hour, seven day a week 
service will be practicable or affordable to 
deal with clinical emergencies. 
There has been much debate regarding the 
dissemination of diagnostic ultrasound 
into professions with little previous expe- 
rience in its performance. With the wide 
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Figure 1 - Portable Duplex ultrasound machine. 



availability of portable Duplex ultrasound 
machines {Figure J), there is a growing 
desire for doctors to be able to perform 
arterial scanning. It seems logical to con- 
sider training vascular specialists in the 
practical application of vascular Duplex 
scanning to cover this 'out-of-hours' short- 
fall. Some work has been done in defining 




Figure 2 - Real time Doppler frequency spectrum. 
Time is along the x axis and Doppler shift frequen- 
cies (expressed in units ofvelocitg) are along the g 
axis. The relative intensities of Doppler shift fre- 
quencies are depicted hg using varging grag scales. 
A lighter pixel represents a higher relative inten- 
sitg at that particular frequencg and time. 



how much practical experience is required 
to attain competence in ultrasound scan- 
ning (4) . Simulation based training in ul- 
trasound is beginning to develop, however 
there is as yet no development for vascular 
ultrasound (5). 

Skills associated with vascular ultrasound 
include using B mode as well as colour cod- 
ed ultrasound. However, quantifying arte- 
rial stenosis requires analysis of the Dop- 
pler spectrum displayed by selective sam- 
pling from the arterial stream by pulsed 
wave Doppler ultrasound [Figure 2). There 
is currently no accepted validated and reli- 
able assessment tool for duplex assessment 
of arterial stenosis. There maybe a need to 
objectify credentialing, as is the case with 
other vascular skills (6). The use of stan- 
dardized tests, perhaps simulation based, 
may be advantageous over current assess- 
ments that are performed on patients. 
Arterial stenosis detection. There are a va- 
riety of skills that need to be learnt before 
a trainee is proficient in arterial stenosis 
detection. The operator needs to under- 
stand how surrogate physiological markers 
correspond to degree of stenosis. Also, the 
operator needs to optimize the Duplex ma- 
chine settings for Color coded Doppler and 
Doppler Spectral analysis. Lastly, direct lu- 
minal measurements are used to corrobo- 
rate findings. 

The physiological basis of stenosis assess- 
ment using Doppler spectral analysis re- 
lates to the physics of blood flow through 
a stenosis. Quantitatively, the amount of 
blood flowing through a stenosis (Q) is de- 
scribed by the following equation: 

Q = vxA 

Where v = velocity and A = cross sec- 
tional area. 

Area reduction in a stenosis is proportion- 
ate to velocity increase. This is true until 
a haemodynamically significant stenosis 
is reached, following which trickle flow 
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ensues (Figure 3). Hence, the percentage 
stenosis can be calculated from the ratio 
of peak systolic velocities at and adjacent 
to the site of stenosis (peak systolic veloc- 
ity ratio; PSVR; Table 1). Previous in-vitro 
work has correlated angiographic diameter 
reduction to PSVR (7). 
As practitioners become more skilled at us- 
ing the ultrasound machine, they will be- 



Table 1 - Sensitivity and Specificity for Peak Systolic 
Velocity Ratio (PSVR) for detection of stenosis. From 
Rankeatal. (1992). 



Stenosis 
(%) 


PSVR 


Sensitivity 
(%) 


Specificity 
(%) 


>30 


> 1.6 


90 


91 


>40 


>2.1 


84 


92 


> 50 


>2.4 


87 


94 


> 60 


>2.9 


84 


91 


> 70 


>3.4 


91 


98 


> 80 


>4.0 


90 


98 


> 90 


> 7.0 


88 


97 



come more adept at optimizing the imaging 
both for colour and pulsed wave modes. 
For colour coded imaging, Duplex settings 
should be adjusted to optimise: frame rate, 
velocity range, colour wall filter, colour 
persistence, colour gain, colour box width. 
For spectral readings, the following setting 
should be optimised: sample volume, veloc- 
ity range, Doppler angle and need for beam 
steering, wall thump filter setting. 
An additional method used to confirm the 
stenosis assessment is the diameter reduc- 
tion at the stenosis. Here the diameter at 
the stenosis is expressed as a percentage of 
the diameter of the disease free artery. 
Assessment of procedural skills. Previously, 
the assessment of technical skills, for in- 
stance in arterial stenosis detection, has 
been limited to the use of logbooks record- 
ing the type and numbers of procedures 
performed and global assessments by train- 
ers through observation (8). 
While these are important evidences of 



Heart, Lung and Vessels. 2014, Vol. 6 



DUOSATS validation 



Duplex Ultrasound Objective Assessment of Technical Skill (DUOSATS) 




SCORING 


DOMAIN 


1 


2 


3 


4 


Patient 
positioning 


Does not 
position patient 
appropriately 


Positions patient 
appropriately 






Transducer 
selection 


Incorrect 
transducer 


Correct 

transducer with 
prompting 


Correct transducer 
without prompting 


Correct transducer without 
prompting with optimum 
frequency selected 


Ultrasound 
coupling gel usage 


None/too much 


Correct amount 






Acquires images 
in B mode 


No attempt made 


Attempt made at B 
mode imaging 




Appropriate depth 
of field, TGC 
and focus setting 
Appropriate gain control? 


Evaluation 
of stenosis 
in colour 


No 


Yes 




Good colour, filling with no 
colour bleed. Appropriaste 
velocity range selected with 
no colour aliasing 
coloor aliasing 


Spectral Doppler 




Appropriate gate 
position and size 


Appropriate gate 
position and size. 
Appropriate 
velocity range, 
baseline and 
inversion settings 


Appropriate gate position 
and size. 

Appropriate velocity range, 

baseline and inversion 

settings. 

No aliasing seen 


Doppler 
Angle 


>60 degree angle 
Doppler angle to 
direction of flow 




<60 degree 
Doppler angle to 
direction of flow in 
centre stream 


< 60 degree Doppler angle 
to direction of flow in centre 
stream 


Assessment 
ofPSVR 


No pre and 
post PSV 
measurements 


Pre/post stenosis 
PSV measurement 
too close to 
stenosis 




PSV measurement 
taken in stenosis jet 
and 4 diameter 
lengths away 


Calculation 
of diameter 
reduction 
at strenosis 


Colour Doppler 
used to assess 
lumen with 
appropriate beam 
steering 


Colour Duplex 
used to assess 
lumen in 
longitudinal 
section with 
appropriate beam 
steering 


Colour Duplex used 
to assess lumen 
in longitudinal 
section with 
appropriate 
beam steering at 
appropriate point 
in cardiac cycle 


Colour Duplex used to assess 
lumen in cross section with 
appropriate beam steering at 
appropriate point in cardiac 
cycle. Good attempt made 
to make an un-diseased 
and actual diameter 
measurement of vessel 


Calculation of 
degree of strenosis 


No attempt made 




With 20% error of 
correct stenosis 


Correct decile calculated for 
degree of stenosis 


Reporting 


Poor legibility, 
comments not 
relevant 


Legible, comments 
not relevant 


Legible report 
with appropriate 
comments on PSVR 


Legible report with 
appropriate comments on 
PSVR, diameter reduction 
and conclusions 



Figure 4 - DUOSATS 
marking scheme. 
DUOSATS = duplex ul- 
trasound objective struc- 
tured assessment of tech- 
nical skills. 
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training that should be recorded, they lack 
content validity as they do not provide an 
assessment of the technical ability of the 
individual performing the procedure (9, 
10). The concept of more objective ap- 
proaches to technical skills assessment is 
gaining greater acceptance (11-13). 
In an attempt to improve the validity and 
reliability of assessment, the objective 
structured sssessment of technical skills 
(OSATS) was developed for surgical tasks 
(14). This uses a combination of a check- 
list and global rating scale to assess train- 



ees performing surgical tasks on models. 
It has been shown to be reliable and have 
content and face validity. More recently, 
OSATS has been used in simulated theatre 
environments (15). 

Procedure-specific global rating scales have 
also been developed, i.e. for sapheno-femo- 
ral ligation, long saphenous stripping and 
multiple avulsions (16). 
A valid, reliable and feasible instrument 
to assess duplex arterial stenosis detection 
is required for formative and summative 
assessment as well as revalidation. The 
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96 duplex ultrasound objective structured as- 
sessment of technical skills (DUOSATS) 
was developed by two experienced prac- 
titioners who routinely clinically practice 
and train in these skills (MA and UJ). 
Content analysis along the principles for- 
mulated by Gagne (the conditions of learn- 
ing 1965) was performed. This informed 
the design of DUOSATS whose domains 
are: atient positioning, transducer selec- 
tion, ultrasound coupling gel usage, ac- 
quires images in B mode, evaluation of ste- 
nosis in colour, spectral doppler, doppler 
angle, assessment of PSVR, calculation of 
diameter reduction at stenosis, calculation 
of degree of stenosis, reporting. 
In order for the assessment instrument to 
function as a formative and summative 
assessment, the scoring for tasks was set 
to reflect progression from lower order 
concepts, through to rule learning and fi- 
nally synthesis of rule learning principles 
into higher order problem solving (Fig- 
ure 4). More complex educational objec- 
tives where given proportionately greater 
weight in the overall score to reflect this. 
Pulsatile flow simulator. The Axiom vas- 
cular flow simulator (Axiom Medical ltd, 
London) forces pulses of blood-mimicking 
solution through a simulated blood vessel, 
arranged within a tissue 'phantom'. 
The flow can be regulated in both the for- 
ward and reverse direction. This allows 
any arterial waveform to be easily simu- 
lated by individually adjusting timings and 
flow rates. A simulated variable arterial 
stenosis is adjustable in the simulated ves- 
sel. This stenosis can be imaged in B mode, 
colour and pulsed wave mode ultrasound, 
allowing stenosis evaluation. This simula- 
tor was used to generate a femoral artery 
waveform in a straight vessel phantom 
containing a 75 percent stenosis. 
This study aimed to establish content, con- 
struct and concurrent validity as well as 
reliability of a novel DUOSAT. 



METHODS 

Medical students, junior doctors of varying 
degrees of experience with Duplex ultra- 
sound and Vascular Scientists (also known 
as Vascular sonographers/ technologists) 
and trainees from Imperial College NHS 
Trust were invited to participate in this 
study. Audit of teaching does not require 
formal ethics committee approval in the UK. 
For the purposes of subsequent data analy- 
sis, participants were classified into nomi- 
nal categories on the basis of stenosis de- 
tection experience ('novice', 'intermediate' 
and 'experienced'). A 'novice' had no prac- 
tical experience of stenosis detection, the 
'intermediate' group had performed one to 
20 cases of stenosis detection and the 'ex- 
perienced' group had performed over 20 
cases. 

An instructional video was prepared and 
played to all participants. The video cov- 
ered theoretical and practical principles 
outlined in 1.2 and demonstrated how ar- 
terial stenosis detection is performed using 
PSVR and diameter reduction methods. 
Sections of the video corresponded to do- 
mains of the DUOSATS. 
Following viewing of the video, sufficient 
time was allowed for the participant to 
familiarize themselves with the function- 
ing of the portable Duplex machine used 
for the study (Mindray M7, Shenzhen, 
China). Questions regarding the function- 
ing of the Duplex machine and the simu- 
lator were answered. When participants 
felt they were sufficiently familiar with the 
machine and the task, they were asked to 
assess a pre-defined stenosis (70 % ) in the 
simulated vessel under femoral artery pul- 
satile flow conditions (Figure 5). Partici- 
pants were asked to use both peak systolic 
velocity ratio as well as diameter reduc- 
tion to form their conclusions. No restric- 
tion to the number of measurements was 
made, however a single percentage was 
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Figure 5 - Participant 
measuring simulated ar- 
terial stenosis. 
Duplex scanner is dis- 
playing both Colour and 
Spectral Doppler scan of 
stenosis being measured. 
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taken from participants for the purpose of 
analysis. Video clips of each attempt were 
recorded for later analysis. 
Participants were asked to record their ste- 
nosis assessment via PSVR and diameter 
reduction. 

Percentage error for both assessments were 
calculated against the actual grade of ste- 
nosis. The video clips were evaluated in- 
dependently by four experienced practitio- 
ners in vascular ultrasound ( > fours years 
practical experience) and were scored on 
the DUOSATS tool. For these simulator- 
based assessments, a modified DUOSATS 
was used; this did not count the patient 
positioning and reporting fields (scoring 
guidance detailed in the Figure 4). 
A 'global assessment' rating was also made 
on a four point Likert scale (level 1 repre- 
senting 'Unable to perform the procedure'; 
level 2 representing Able to perform the 
procedure with prompting', level 3 rep- 
resenting Able to perform the procedure 
with minimum prompting'; and level 4 
representing 'Competent to perform the 
procedure unsupervised'). 
A four point scale forces raters to take a 
position and is associated with clearer de- 
cision making (17). 

The four point Likert scale (with the addi- 
tion of 0 signifying not enough data avail- 



able to make assessment) format has been 
extensively used recently in the context of 
procedure -based assessments in the UK In- 
tercollegiate Surgical Curriculum Project. 
The scale has been reported to have a high 
degree of acceptability by assessors and 
trainees (18). 

Demographic data and experience with 
ultrasound and specifically stenosis detec- 
tion were recorded. 

Participants were asked to rate their sat- 
isfaction with the pulsatile flow simulator 
on the five point Likert scale. 
Analysis. Inter-observer reliability between 
the four assessors was determined using 
Cronbach's alpha (a). Non-parametric tests 
were used in all the analyses. Spearman's 
Rank (R) was used to correlate continuous 
variables. 

The Kruskal Wallis test was used to iden- 
tify differences between the subgroups 
tested. Construct validity was assessed by 
comparing DUOSATS scores of expert, in- 
termediate and novice participants. Con- 
current validity was assessed by compar- 
ing percentage error in stenosis detection 
(PSVR and diameter reduction method) 
and overall rating to DUOSATS. SPSS 20 
(IBM corporation) was used in the statisti- 
cal analysis p< 0.05 was considered statisti- 
cally significant. 
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RESULTS 

Demographic data. There were 23 partici- 
pants in total, of which 15 were male (65 % ). 
Content validity. The DUOSAT was devel- 
oped in conjunction with two experienced 
practitioners and trainers in vascular ultra- 
sound and addresses domains considered 
essential in Duplex arterial stenosis detec- 
tion according to the principles of content 
analysis. 

The device was rated by all participants 
on a five point rating scale with one giv- 
en for a poor representation and five for 
an excellent representation of vascu- 
lar ultrasound. The scores ranged from 
3-5 (median 4), the higher scores com- 
ing from the more experienced partici- 
pants, however this difference was not 
significant (Kruskal-Wallis p = 0.813) . 
Construct validity. The average scores for 
both DUOSATS and the 'global' assess- 
ment were used in all subsequent statisti- 
cal analyses. Scores for DUOSATS ranged 
from 18.25 to 33.24 (median 19.6 of a max- 
imum of 45). Calculated PSV ratio's ranged 
from 0.87-4.83 (median 2.21). The percent- 
age stenosis using diameter reduction cri- 



teria ranged from 46-83% (median 70%). 
Study participants were assessed accord- 
ing to previous practical ultrasound experi- 
ence (novice n = 9, intermediate n = 8 and 
experienced n = 6; Table 2). There were no 
significant differences in the DUOSATS 
score (Kruskal Wallis; p = 0.122) or the 
'global' assessment (p = 0.143) across the 
three groups. The groups were subsequent- 
ly analysed according to previous stenosis 
measurement experience. This analysis 
identified significant differences with both 
measures (DUOSATS p = 0.004, 'global' 
assessment (p = 0.006). 
A further analysis of individual DUO- 
SATS domains using stenosis measure- 
ment experience was performed to es- 
tablish construct validity for individual 
items and the results are summarised in 
Table 3. Statistically significant fields were 
acquisition of image in B mode, evalua- 
tion of stenosis using Spectral Doppler 
angle, calculation of diameter reduction 
and calculation of degree of stenosis. 
Concurrent validity. When the two mea- 
sures were correlated, there was a signifi- 
cant correlation between DUOSATS score 
and 'global' assessment scores {Figure 6) 



Table 2 - Demographic data for participants. Ultrasound (US) experience (1: Theoretical; 2: 1-20 cases; 3: >20 
cases). Stenosis detection experience (1:1-10 cases; 2: 10-20 cases; 3: >20 cases). 





Novice 


Intermediate 


Experienced 


Total 


Number 


9 


8 


6 


23 


Age (years) 


25 (22-41) 


33.5 (22-39) 


28.5 (26-36) 


28 (22 - 39) 


Sex (male) 


7 


7 


1 


15 


Designation: 










Medical Student 


5 






5 


Junior Doctor 


4 






5 


Specialist Registrar 




4 




4 


Vase. Scientist/ trainee 




4 


6 


10 


US experience (median; range) 


0; 0-3 


3; 2-3 


3; 3-3 




Stenosis experience (median; range) 


0; 0-1 


3; 2-3 


3; 3-3 
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Table 3 - Breakdown of individual domains within Duplex Ultrasound Objective Structured Assessment of Techni- 
cal SHU (DUOSATS; *p<0.05). Peak Systolic Velocity Ratio (PSVR). 
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DUOSATS domain 


Median (Range) 


p (Kruskal Wallis) 


Transducer selection 


4 (3.25 - 4) 


0.753 


Ultrasound coupling gel usage 


3 (2.5 - 3.75) 


0.444 


Acquires images in B mode* 


2 (1 - 3) 


0.046* 


Evaluation of stenosis in colour 


2.75 (1-4) 


0.193 


Spectral Doppler* 


2.13 (1.25-4) 


0.025* 


Doppler Angle 


1.75 (1 -3.75) 


0.201 


Assessment of PSVR 


1.25 (1-4) 


0.104 


Calculation of diameter reduction at stenosis* 


2(1-4) 


0.009* 


Calculation of degree of stenosis* 


1.38 (1 -3.75) 


0.015* 



(Spearman's rank correlation coefficient; 
R = 0.737, p = 0.000092). The DUOSATS 
score correlated with stenosis measure- 
ment using PSVR criteria (R = 0.46, 
p = 0.03; Figure 7) but not with stenosis 
measurement using diameter reduction, 
criteria (R = 0.4, p = 0.065; Figure 8). 
On multivariate analysis, age (contribu- 



tion is 29.8 % , p = 0.05) and stenosis detec- 
tion experience had a significant effect on 
score (contribution is 43.1%, p = 0.008). 
However, ultrasound experience had no 
effect on the DUOSATS score (contri- 
bution = 1.7% of variance, p = 0.861). 
Reliability. Four independent expert asses- 
sors reviewed the videotape footage of each 



\- 25.00- 
< 

c 
a 

Q 




Average Overall Score (1-4) 



Figure 6 - Scatter plot of 
average overall ('global') 
score versus DUOSATS 
score demonstrating cor- 
relation. 

DUOSATS = duplex ul- 
trasound objective struc- 
tured assessment of tech- 
nical skills. 
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Figure 7 - Scatter plot of 
average D UOSATS score 
versus the error score ste- 
nosis measurement using 
PSV ratios demonstrat- 
ing correlation. 
DUOSATS = duplex ul- 
trasound objective struc- 
tured assessment of tech- 
nical skills; PSV = peak 
sgstolic velocity ratio. 



25.00 

DUOSATS score 



participant performing stenosis measure- 
ment. Assessors were blinded to the iden- 
tity of the study participant. Inter-observer 
reliability for DUOSATS and 'global' as- 
sessment scores were high (Cronbach's 
alpha = 0.971 and 0.957 respectively). 
Sensitivity and specificity. A Receiver Op- 
erator Characteristics curve (ROC; Figure 



9) was plotted for the sensitivity and speci- 
ficity of the test using the scores of those 
experienced in stenosis detection to deter- 
mine the cut point in determining compe- 
tence for this procedure. The area under 
the curve was 0.895. For various cut points 
the specificity and sensitivity is shown in 
Table 4. It was decided the cut-point should 
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Figure 8 - Scatter plot 
of average DUOSATS 
score versus error score 
for stenosis calculation 
using diameter reduction 
criteria. 

DUOSATS = duplex ul- 
trasound objective struc- 
tured assessment of tech- 
nical skills. 
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Figure 9 - ROC curve 
demonstrating the corre- 
lation hetween true posi- 
tive versus false positive 
rates. Using these char- 
acteristics, an appropri- 
ate cut point for the eval- 
uation were determined. 
ROC = receiver operator 
characteristics curve. 



have a high specificity (i.e. less experienced 
participants were unlikely to be consid- 
ered competent) trading off for a slightly 
reduced sensitivity for this test (more ex- 
perienced participants may not achieve 
competence in this test. The cut point was 
set at 22 (out of a possible 34). 

DISCUSSION 

Regarding assessment instruments, Cook 
and Beckman discuss that "validity is not 
a property of the instrument, but of the 
instrument's scores and their interpre- 
tations" (19). Both DUOSATS and the 
'global' assessment were able to differenti- 
ate participants according to their experi- 
ence of Duplex arterial stenosis detection 
(DUOSATS: p = 0.004, 'global' assessment; 
p = 0.006). This suggests good construct 
validity. Importantly, this discriminating 
ability did not extent to previous ultra- 
sound experience not specifically related 
to arterial stenosis detection (DUOSATS: 



p = 0.122 'global' assessment p = 0.143) 
across the three groups, suggesting that 
DUOSATS is sensitive to specific skills re- 
lated to the task at hand and has content 
validity. It may be suggested that 'global' 
assessment is an acceptable alternative to 
DUOSATS, however the limitation is the 
lack of content validity and unstructured 
unsystematic approach. DUOSATS score 
correlated with stenosis measurement us- 
ing PSVR criteria (R = 0.46, p = 0.03) in- 
dicating concurrent validity with an objec- 
tive 'end product' style assessment. 
A potential criticism of DUOSATS is its 
development by only two people. However, 
the DUOSATS domains cover a wide range 
of skills related to the task and multiple do- 
mains show significant discriminating abil- 
ity to level of experience. The DUOSATS 
domains best able to discriminate between 
novice, intermediate and experienced users 
were found to be: acquisition of image in B 
mode, evaluation of stenosis using spectral 
doppler angle, calculation of diameter re- 
duction and, calculation of degree of steno- 
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102 Table 4 - Specificities and Sensitivities associated with various cut points for the Duplex Ultrasound Objective As- 

sessment of Technical Skill (DUOS ATS) score. 



Cut Point 


True Positive Rate 
(Sensitivity) 


True Negative Rate 
(1 - Specificity) 


17.2500 


1.000 


1.000 


18.3750 


1.000 


.800 


18.7500 


1.000 


.667 


19.1250 


1.000 


.533 


19.3750 


1.000 


.400 


19.6250 


.857 


.333 


20.0000 


.714 


.267 


20.3750 


.714 


.200 


21.0000 


.714 


.133 


21.8750 


.714 


.067 


24.0000 


.571 


.067 


28.8750 


.571 


.000 


32.6250 


.429 


.000 


34.2500 


.000 


.000 



sis. These domains should be most closely 
followed when using the DUOSATS in 
formative assessment. Careful feedback on 
these fields may focus and hasten training. 
ROC data suggests that a score of 22 (out 
of a possible 34) has high sensitivity and 
specificity, sufficient for high stakes assess- 
ment on the pulsatile flow simulator. We 
believe that it is reasonable to increase the 
pass mark to 25 (out of a possible 40) for 
the 'full' DUOSATS scoring to be used in 
patient assessment. This data facilitates 
standard setting in assessing Duplex ar- 
terial stenosis evaluation not subject to 
the 'halo' effect encountered in relative 
or norm referenced assessment tools. Al- 
though this methods of standard setting 
is itself technically 'norm referenced', we 
feel that introduction of this skill to a new 



group of practitioners should not be asso- 
ciated with a decline in overall standards. 
This approach potentially goes some way 
to satisfy the fit for purpose, supported by 
research, easy implement standard of as- 
sessment (20). Further validation of the 
full DUOSATS on patients is underway. 
We have demonstrated content and concur- 
rent validity. Inter-rater reliability amongst 
experienced assessors was high, comparing 
favorably with other objective structured 
technical skills assessments (14, 15, 21). 
Participants rated the simulator highly in 
terms of satisfaction. This compares fa- 
vourably with other technical skills simu- 
lators (22-24). Investigators in a number 
of technical skills simulation disciples have 
reported transference of skills from bench 
top simulators to a human cadaver model 
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(25) as well as operating theatre environ- 
ments (26, 27). Further work is required, 
in the form of a trial, to assess transfer- 
ence of skills learnt using this pulsatile 
flow simulator to scanning on patients. 
Further development in phantom design 
may be desirable to provide more challeng- 
ing anatomy, for instance the obese patient 
and arterial wall calcification. Validation of 
the DUOSATS system for simulator train- 
ing potentially represents an early yet im- 
portant milestone in the development of 
training and assessment in Duplex arterial 
assessment. Validation of the DUOSATS 
for patient scanning remains to be as- 
sessed. We envisage potential difficulties in 
terms of patient variability, a problem not 
encountered in 'standard' simulated sce- 
narios. This may potentially be overcome 
by the use of statistical process control 
techniques and learning curve assessment 
rather than a single point assessment. 

Limitations 

This study only used a single severity of 
stenosis for validation of the OSAT. This 
was chosen to be 70 percent as this degree 
of stenosis is clinically significant. We have 
not established validity using different de- 
grees of stenosis. Further work should take 
this into consideration and also have in- 
creased number of participants to compen- 
sate for the increased variability of errors 
when considering different degrees of ste- 
nosis. The validation has been performed 
for a modified DUOSATS for simulator 
training. Further correlation with the full 
DUOSATS for actual patient testing to in- 
clude patient positioning, will need sepa- 
rate assessment. 



CONCLUSION 

We have developed a novel objective struc- 
tured assessment of technical skills for du- 



plex assessment of arterial stenosis. The 
modified DUOSATS for simulator training 
and assessment was validated. 
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